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DEVELOPMENT  ACTIVITIES  LOCATOR  AND  ASSESSMENT  METHOD 

SUMMARY 


THE  PROJECT  PURPOSE: 

To  improve  the  process  by  which  Combined  Joint  Task  Force-  Horn  of  Africa  (CJTF-HOA) 
locates  Civil  Affairs  (CA)  teams,  selects  and  resources  projects,  and  assesses  projects’ 
contributions  to  campaign  objectives  and  local  national  people’s  well-being. 

THE  PROJECT  SPONSOR: 

The  problem  was  initially  posed  by  the  fonner  CJ-5  of  CJTF-HOA,  Captain  Owen  Travis,  US 
Navy.  Dr.  Shilling  formalized  the  problem  statement  and  Captain  Pete  Haynes,  US  Navy,  then- 
current  CJ-5,  signed  the  analysis  request  that  formally  initiated  the  study. 

THE  PROJECT  OBJECTIVES  were  to: 

(1)  Automate  and  improve  the  decision  matrix  used  to  recommend  the  placement  of  CA 
teams. 

(2)  Identify  the  principles  of  effective  assistance  and  integrate  those  principles  into 
“worksheets”  used  for  assessing  needs,  comparing  and  selecting  projects,  drafting  project 
proposals  and  plans,  and  guiding  execution  and  assessment  of  projects. 

(3)  Inject  assessment  methodology  into  all  aspects  of  CA  projects:  planning,  execution,  and 
follow-up.  Develop  “metrics  menus”  to  assist  teams  in  the  field  in  selecting  appropriate  metrics. 

THE  SCOPE  OF  THE  PROJECT: 

Effort  is  limited  to  improving  the  methods  used  to  locate,  plan,  execute,  and  assess  CA  activities 
in  CJTF-HOA’s  area  of  operations. 

THE  MAIN  ASSUMPTION: 

Assistance  projects  generate  two  types  of  effects — development  effects  and  influence  effects — 
and  lasting  influence  effects  are  unlikely  unless  beneficiaries  receive  a  lasting  development 
effect. 

THE  PRINCIPAL  FINDINGS  are: 

Considerable  consensus  exists  in  the  development  community  on  principles  that  make  assistance 
more  effective,  and  military  organizations  doing  similar  work  can  improve  effectiveness  by 
applying  these  principles. 

THE  PRINCIPAL  RECOMMENDATIONS  are: 

(1)  The  inclusion  of  items  designed  to  ensure  local  national  ownership,  capacity-building, 
and  sustainability  of  development  projects  in  project  nomination  forms  greatly  increases  the 
likelihood  of  a  significant  and  long-lasting  development  effect. 
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(2)  A  lasting  development  effect  greatly  increases  the  likelihood  of  a  satisfactory  and  long- 
lasting  influence  effect  that  changes  the  attitudes  and  behaviors  of  local  national  people  toward 
American  organizations. 

THE  PROJECT  EFFORT  was  conducted  by  Adam  Shilling,  PhD. 

COMMENTS  AND  QUESTIONS  may  be  sent  to  the  Director,  Center  for  Anny  Analysis, 
ATTN:  CSCA-OA,  6001  Goethals  Road,  Suite  102,  Fort  Belvoir,  VA  22060-5230. 
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1  INTRODUCTION 

1.1  Problem  Statement 


Combined  Joint  Task  Force-  Horn  of  Africa  (CJTF-HOA)  is  not  satisfied  with  the  methods  they 
currently  use  to  plan  for  and  assess  development,  relief,  and  other  non-lethal  activities.  The 
command  is  concerned  that  these  activities,  usually  performed  by  attached  Civil  Affairs  (CA) 
teams,  are  not  as  effective  as  they  could  be  and  that  assessment  of  these  activities  can  be 
improved. 

A  great  deal  of  literature,  including  news  reports,  editorials,  academic  research  and  official 
reports  from  governments,  non-governmental  organizations  (NGOs),  and  international 
organizations,  states  or  implies  thatU.S.  Government  (USG)  development  and  stabilization 
initiatives  often  fail  to  achieve  their  objectives.  Some  examples  include: 

Commission  on  Wartime  Contracting  (20 11).  Report  to  Congress:  Transforming 
Wartime  Contracting. 

William  Easterly  (2006).  The  White  Man  s  Burden:  Why  the  West’s  Effort  to  Help  the 
Rest  Have  Done  So  Much  III  and  So  Little  Good. 

Non-Governmental  Organization  (NGO)  Consortium  (2011).  White  Paper:  Being 
Smart  About  Development  in  Afghanistan. 

Oxfam  America  (2008).  Smart  Development:  Why  U.S.  Foreign  Aid  Demands  Major 
Reform. 

These  programs  often  fail  to  achieve  their  objectives  due  to  poor  planning  or  execution,  and 
therefore,  waste  American  and  international  resources.  The  following  report  is  an  attempt  to 
detennine  what  must  be  done  to  make  Department  of  Defense  (DoD)  initiatives  more  successful. 

1.2  Key  Definitions 


The  reader  requires  some  key  definitions: 

-  Assessment,  in  Civil  Affairs  and  development  agency  usage,  refers  to  processes  of 
gathering  and  organizing  information  about  the  civil  environment  and  the  needs  of 
people  in  it.  It  is  somewhat  analogous  to  intelligence  preparation  of  the  battlefield 
(IPB). 

Assessment  also  refers  to  the  processes  that  are  used  to  evaluate  the  effectiveness  of 
the  U.S.  Government’s  activities  and  to  gauge  progress  toward  campaign  objectives. 

Beneficiaries  are  the  persons,  organizations,  or  institutions  that  benefit  from  a  project 
or  engagement.  Use  of  the  term  within  the  study  usually  will  NOT  refer  to  the  USG 
or  U.S.  military  although  projects  or  engagements  are  hoped  to  bring  collateral 
benefit  to  American  organizations. 

A  development  effect  is  a  change  in  beneficiaries’  well-being  or  capacities  due  to  the 
project  or  an  engagement.  Development  effects  provide  local  national  beneficiaries 
with  reasons  to  cooperate  with  USG  and  U.S.  military. 
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An  influence  effect  is  the  change  in  local  national  attitudes  or  behaviors  toward  the 
U.S.,  USG,  U.S.  military  or  allies  resulting  from  a  project  or  an  engagement. 
Influence  effects  are  usually  those  that  directly  impact  campaign  plan  objectives,  but 
are  not  those  that  interest  local  national  actors. 

Stakeholders  are  people  or  organizations  that  perceive  themselves  as  having  an 
interest  in  a  project.  All  beneficiaries  are  stakeholders,  but  not  all  stakeholders  are 
beneficiaries. 

I  would  like  to  highlight  two  key  points. 

First,  “assessment”  has  two  definitions.  The  first,  commonly  used  in  Civil  Affairs  doctrine,  at 
the  United  States  Agency  for  International  Development  (USAID),  and  in  the  development 
community,  refers  to  activities  undertaken  to  understand  the  operational  environment  and  the 
needs  of  the  people  in  it,  before  action  is  taken.  This  affects  the  selection  of  the  best  among 
several  possible  development  interventions.  This  type  of  assessment  is  analogous  to  intelligence 
preparation  of  the  battlefield  (IPB).  It  also  provides  the  baseline  to  which  subsequent 
assessments  are  compared. 

The  other  meaning,  most  common  in  the  DoD,  refers  to  attempts  to  judge  the  effect  of  actions 
already  taken  under  a  campaign  plan,  relative  to  the  goals  of  the  campaign. 

Of  course,  if  one  imagines  a  circular  problem-solving  method  in  which  an  assessment  is 
followed  by  an  action,  then  another  assessment,  in  a  cycle,  “before”  and  “after”  assessments  are 
seen  to  have  the  same  nature. 

Second,  and  more  important,  DoD-funded  development  activities  usually  have  objectives  that 
fall  into  two  distinct  categories.  The  immediate  objective  of  a  development  activity  is 
development — an  improvement  in  the  well-being  of  the  affected  populace.  The  study  refers  to 
these  improvements  as  “development  effects.” 

On  the  other  hand,  DoD  agencies  are  motivated  to  fund  development  activities  for  some  purpose 
other  than  development.  Typically,  DoD  projects  are  designed  to  influence  attitudes  or  behaviors 
of  the  affected  populace  relative  to  a  military  objective.  The  study  will  refer  to  these  as 
“influence  effects.” 

The  latter  are  what  the  military  hopes  to  get,  relative  to  the  campaign  plan,  from  executing  the 
project,  and  the  former  are  what  the  local  people  hope  to  get  from  cooperating  with  U.S.  forces. 
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1.3  Improving  Processes 


CA  Methodology 


Assess 


Decide 


Develop  and 
Detect 


Deliver 


Evaluate 


Transition 


Proposed  Methodology 


Assessment  determines  needs. 


Civil  Affairs  Assessment 


Alternative  projects  and 
conflicting  needs  compete  for 
selection.  Most  effective  projects 
given  priority.  Identify  effects  on 
campaign. 


Effective  Project 
Worksheets 


Selected  project  plan(s)  fully 
developed. 


Project  executed. 


Assessment  Plan 
Statement  of  Work 


Project  Execution 


Assessed  during/after  execution. 


Assessment 


Got  Ownership? 


Hand  off  to  HNG*  or  beneficiaries 

*Host  nation  government 


Figure  1.  Improving  Processes 

On  the  left  side  of  the  chart  in  Figure  1  is  the  “CA  methodology”  found  in  Field  Manual  (FM)  3- 
57  Civil  Affairs  Operations  (2011).  On  the  right  is  a  breakdown  of  that  methodology  to 
illuminate  how  proposed  improvements  to  planning  and  assessment  methods  integrate  into,  but 
do  not  change,  CA  doctrine.  The  red  box  indicates  where  this  section  of  the  study  is  seeking  to 
refine  the  CA  methodology. 

1.4  Assumptions 


The  study’s  assumptions  were: 

Assistance  projects  generate  two  types  of  effects — development  effects  and  influence 
effects. 

Neither  effect  can  be  assumed  to  be  positive — U.S.  actions  have  unintended  effects, 
which  can  lead  to  unexpected  outcomes. 

Lasting  influence  effects  are  not  likely  unless  beneficiaries  receive  a  lasting 
development  effect. 

Assessment  works  best  when  integrated  throughout  project  planning  and  execution. 

Principles  of  effective  assistance  are  known — outside  the  DoD.  They  are  often 
difficult  to  execute. 
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Project  planning  that  adheres  to  principles  of  effective  assistance  both  increases  the 
likelihood  of  a  positive  effect  and  maximizes  the  magnitude  of  that  effect — whether  it 
be  a  development  effect  or  an  influence  effect. 

The  assumptions  speak  for  themselves,  but  I  would  like  to  amplify  the  third.  In  my  experience 
as  a  civil-military  operations  officer  for  a  brigade  combat  team  in  Iraq,  I  found  many  military 
personnel  to  be  focused  on  the  influence  effect  for  which  a  development  activity  is  pursued.  For 
them,  the  purpose  of  the  activity  is  to  generate  this  influence  effect,  while  the  development  effect 
is  of  almost  negligible  importance. 

I  argue  that  this  attitude  is  shortsighted.  The  local  people  have  no  inherent  reason  to  support 
U.S.  objectives,  but  every  incentive  to  cooperate  with  the  United  States  to  make  their  lives  better. 
This  cooperation  is  an  opportunity  to  influence  them. 

Additionally,  planning  development  activities  to  produce  a  “win-win”  situation  has  the  potential 
to  make  development  activities  part  of  a  transfonnational  relationship,  rather  than  merely  a 
transactional  exchange.  In  this  paradigm,  Americans  observe  local  people’s  actual  needs,  and 
plan,  in  cooperation  with  them,  to  build  sustainable  improvements  to  their  lives  according  to  an 
agenda  that  local  people  choose  for  themselves. 

Failing  the  creation  of  a  transfonnational  relationship,  at  a  minimum,  a  DoD-funded 
development  activity  is  a  transaction  between  local  people  and  the  U.S.  military  where  the 
United  States  trades  the  project  for  a  desired  behavior  from  the  local  people.  Provided  the  locals 
agree  to  this  transaction  explicitly,  it  is  difficult  to  get  them  to  agree  to  maintain  the  behavior  we 
desire  if  the  project  is  no  longer  delivering  the  benefits  they  expect. 

Therefore,  even  if  our  development  activities  are  merely  transactional,  we  still  want  them  to  have 
a  reasonable  lifespan,  to  be  “sustainable.” 

1.5  Essential  Elements  of  Analysis  and  Objectives 


The  following  Essential  Elements  of  Analysis  (EEAs)  and  Measures  of  Effectiveness  (MoEs)  are 
part  of  the  researcher’s  attempt  to  ensure  the  quality  of  the  study  and  to  ensure  it  achieves  its 
objectives. 

Objective  1 :  More  effective  decision  matrix  for  placement  of  CA  teams,  which  is 
automated  for  ease  of  use. 

o  EEA  1.1:  Can  automated  matrix  be  used  by  operational  planners? 

■  MoE:  Yes/No. 

o  Product:  Automated  Decision  Matrix  in  Excel. 

Objective  2:  Effective  Projects. 

o  EEA  2.1:  What  is  the  strength  of  consensus  in  the  development  community  for 
key  “principles  of  development”? 

■  MoE:  Strength  of  consensus  on  synthesis  matrix. 

o  EEA  2.2:  Can  principles  with  strong  consensus  be  integrated  into  project  planning 
and  selection? 

■  MoE:  Yes/No. 
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o  Product:  Effective  Projects  Worksheets. 

Objective  3:  Integrated  Assessment  Method. 

o  EE  A  3. 1 :  Do  complete  assessment  plans  become  a  mandatory  part  of  project 
planning  before  project  execution? 

■  MoE:  Yes/No. 

o  EEA  3.2:  Are  relevant  engineering/safety  standards  included  in  statements  of 
work  (SOWs)? 

■  MoE:  Yes/No. 

o  EEA  3.3:  Are  assessment  visits  performed  during  and  after  execution  to  ensure 
project  delivers  intended  benefits  over  time? 

■  MoE:  Yes/No. 

o  Product:  Assessment  Template. 

o  Product:  “Metrics  Menu.” 
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2  CIVIL  AFFAIRS  LOCATION  DECISION  MATRIX 

2.1  Automation  of  Decision  Matrix 


Objective  1  requires  improvement  to  an  existing  decision 
matrix. 

—  Mr.  Griffith  and  Mr.  Gellerman  have  designed  an  automation  of  the 
existing  matrix  that  improves  rigor  and  is  user  friendly. 


Original, 

manual 

decision 

matrix. 


Automated 
matrix.  More 
rigorous  and  user 
friendly. 


T 


Figure  2.  Automation  of  Decision  Matrix 

The  first  objective  sought  to  automate  and  improve  the  existing  decision  matrix  used  by  the  CA 
battalion  for  assigning  CA  teams  to  a  location  within  the  Combined  Joint  Task  Force  (CJTF)  area 
of  responsibility  (AOR)  in  January  2012.  Several  CAA  analysts  with  programming  experience 
contributed  to  an  Excel- based  upgrade.  Mr.  Griffith,  who  was  deployed  to  the  CJTF  staff  at  that 
time,  spearheaded  this  effort  and  produced  an  automated  version  of  the  CA  battalion’s  matrix. 
The  feedback  from  the  CA  battalion  was  positive,  but  the  tool  created  does  not  explicitly 
consider  the  desires  of  key  individuals,  such  as  the  CJTF  commander  and  U.S.  ambassadors  in 
candidate  countries. 

One  Senior  Executive  Service  (SES)  leader  at  the  Pentagon,  when  briefed  on  the  study,  criticized 
the  choice  of  criteria  the  CA  used.  He  pointed  out  that  the  criteria  in  the  matrix  determined  here 
it  was  easiest  for  the  CA  to  work,  and  not  where  the  CA  was  needed  to  improve  U.S.  influence  or 
access. 
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2.2  Automated  Decision  Matrix  Sample  Output 


Advantages: 

-User  friendly. 

-Easy  data  entry. 

-Easy  to  vary  weights. 

-Easy  to  answer  “what  ifs” 
-Simplified  sensitivity  analysis. 
-Visually  intuitive  output. 
-Contribution  of  different  criteria 
to  recommendations  broken  out. 
-Meaningful  continuous  variables. 
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CANDIDATE  LOCATION 


Embassy  Distance 

■  Arc  Distance 
Road  Quality 

□  Medical  Quality 
Mil  -  Mil  Access 

■  KLE  Access 
Market  Access 
Population  Access 

■  Airfield  Access 


Figure  3.  Automated  Decision  Matrix  Sample  Output 

Figure  3  above  displays  a  sample  output  of  the  improved  matrix.  It  is  more  user-friendly  and 
allows  analysts  to  easily  vary  weights  of  criteria  in  order  to  answer  questions  or  conduct 
sensitivity  analysis.  The  output  is  intuitive:  the  tallest  bar  is  the  “best”  option,  and  the 
contribution  of  each  criterion  to  recommended  solutions  is  easily  distinguished  by  the  colors. 
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3  PRINCIPLES  OF  EFFECTIVE  ASSISTANCE 

3.1  Project  Design  Improvement 


I  approached  this  study  with  experience  as  a  brigade  civil-military  operations  officer,  and  I 
thought  the  apparent  problem  with  development  activities  was  not  merely  the  inability  to 
document  the  progress  we  were  making.  Rather,  I  believed  that  a  given  project  achieved  less 
than  its  potential  because  our  project  design  procedures  could  be  improved. 

A  survey  of  literature  provided  evidence  for  this  belief.  Projects  were  less  effective  than  their 
potential  due  to  poor  project  design.  Thus,  the  task  was  not  merely  to  improve  the 
documentation  of  progress  via  assessment,  but  to  improve  the  selection  and  planning  of  projects 
to  be  more  effective  relative  to  both  desired  development  and  influence  effects. 

Under  the  assumption  that  lasting  influence  effects  are  unlikely  without  lasting  development 
effects,  most  of  this  report  is  about  achieving  lasting  development  effects. 

Influence  effects  are  achieved  by  the  placement  of  a  developmentally-successful  project  among 
the  right  group  of  local  people  at  the  right  time. 

3.2  Project  Planning  and  Selection 


CA  Methodology 


Assess 


Decide 


Develop  and 
Detect 


Deliver 


Evaluate 


Transition 


Proposed  Methodology 


Assessment  determines  needs. 


Civil  Affairs  Assessment 


Alternative  projects  and 
conflicting  needs  compete  for 
selection.  Most  effective  projects 
given  priority.  Identify  effects  on 
campaign. 


Effective  Project 
Worksheets 


Selected  project  plan(s)  fully 
developed. 


Assessment  Plan 
Statement  of  Work 


Project  executed. 


Project  Execution 


Assessed  during/after  execution. 


Assessment 


Got  Ownership? 


[Hand  off  to  HNG*  or  beneficiaries 


*Host  nation  government 


Figure  4.  Project  Planning  and  Selection 
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Project  planning  and  selection  requires  tools  to  assess  needs,  prioritize  needs,  ensure  project 
planning/proposal- writing  are  in  keeping  with  best  practice,  and  project  execution  and 
assessment  are  integrated  from  project  initiation. 

This  section  of  the  report  deals  with  refinement  of  the  “Decide”  portion  of  the  CA  methodology 
shown  in  Figure  4  above,  as  indicated  by  the  red  box.  Project  planners  evaluate  project 
nominations  using  the  principles  discussed  below  to  detennine  which  projects  are  likely  to 
succeed  in  achieving  the  desired  development  effect. 

The  second  objective  sought  to  refine  this  portion  of  the  CA  methodology  by  identifying  the 
principles  of  effective  assistance  and  applying  them  to  the  project  nomination  process  in  order  to 
select  and  plan  the  projects  most  likely  to  succeed. 

A  brief  overview  of  the  CA  methodology  may  be  helpful.  The  steps  are: 

Assess — Conduct  a  baseline  assessment  of  the  civil  environment  and  development  needs. 
It  should  answer  such  questions  as: 

o  What  is  the  current  state  of  relevant  civil  matters?  (CA  assessment  checklists 
exist  and  are  a  good  place  to  start.) 

o  What  are  local  people’s  needs  (relative  to  objective  standards  from  appropriate 
organizations)? 

o  What  are  local  people’s  self-perceived  needs? 
o  What  are  local  people’s  preferences? 

Decide — Generate  alternative  project  proposals  and  determine  those  to  be  funded. 
Develop  and  Detect — Develop  comprehensive  project  and  assessment  plans. 

Deliver — Execute  the  project. 

Evaluate. 

Transition. 

CA  manuals  contain  some  useful,  detailed  assessment  templates  for  an  initial  assessment  of  the 
state  of  civil  affairs  in  a  given  area. 

In  addition  to  the  assessment  templates,  it  is  essential  to  ask  the  people  what  they  want,  and  what 
they  feel  they  need,  and  compare  these  with  trained  observation  on  what  they  might  actually 
need  relative  to  published  standards.  These  three  things  might  not  agree. 

This,  however,  begins  a  conversation  between  U.S.  soldiers  and  local  people  that  might  develop 
into  a  project.  The  conversation  is  necessary  because  soldiers  are  limited  in  that  they  do  not 
understand  the  culture  or  problems  local  people  face.  Local  people  are  limited  in  their  vision. 
They  often  know  little  of  the  outside  world,  little  of  the  potential  of  their  lives  to  change  for  good 
or  ill,  and  may  not  understand  their  own  assumptions  about  their  lives  that  hold  them  back. 

The  conversation  mitigates  the  shortcomings  of  both  groups,  and  may  also  begin  a 
transformational  relationship  that  might  achieve  the  influence  effect  that  U.S.  commanders 
desire. 

3.3  Determine  Principles.  What  makes  projects  effective? 


Given  that  development  is  not  DoD’s  area  of  expertise,  consultation  with  organizations  with 
development  expertise  may  help  DoD  improve  its  activities.  I  did  this  by  finding  written 
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statements  from  as  many  of  these  organizations  as  I  could  and  comparing  them  in  a  qualitatively 
rigorous  fashion. 

This  literature  review  reveals  apparent  consensus  among  relevant  organizations  on  the  principles 
of  effective  assistance.  Each  group  had  a  different  emphasis  or  made  statements  in  different 
ways,  but  they  still  agreed. 

To  demonstrate  this  consensus,  I  used  a  qualitative  synthesis  technique.  I  started  with  the  “nine 
principles  of  development”  published  by  the  U.S.  Agency  for  International  Development 
(USAID)  and  compared  other  organizations’  statements  to  USAID’s.  Then  I  observed  and 
highlighted  areas  where  consensus  exists. 

3.4  USAID’s  “Nine  Principles” 


I  based  the  comparison  on  USAID’s  “nine  principles  of  development”  as  defined  and  explained 
in  USAID  documents.  These  are: 

Ownership :  Build  on  the  leadership,  participation,  and  commitment  of  a  country  and  its 
people. 

Capacity-Building :  Strengthen  local  institutions,  transfer  technical  skills  and  promote 
appropriate  policies. 

Sustainability :  Design  programs  to  ensure  their  impact  endures. 

Selectivity.  Allocate  resources  based  on  need,  local  commitment,  and  foreign  policy 
interests. 

-  Assessment.  Conduct  careful  research,  adapt  best  practices,  and  design  for  local 
conditions. 

-  Results:  Focus  resources  to  achieve  clearly  defined,  measurable,  and  strategically-focused 
objectives. 

-  Partnership :  Collaborate  closely  with  governments,  communities,  donors,  NGOs,  the 
private  sector,  international  organizations,  and  universities. 

Flexibility :  Adjust  to  changing  conditions,  take  advantage  of  opportunities,  and  maximize 
efficiency. 

Accountability:  Design  accountability  and  transparency  into  systems  and  build  effective 
checks  and  balances  to  guard  against  corruption. 

I  found  as  many  concise  statements  of  principles  from  different  development  organizations  as  I 
could  and  compared  them  to  USAID’s  principles  in  a  matrix  (Figure  5  below). 

The  following  section  identifies  and  briefly  describes  the  organizations  in  the  analysis,  and  then 
provides  the  name  of  each  respective  statement  of  principles. 

3.5  Statements  of  Principle 


Fiterature  review  began  with  any  development  or  relief  organization  that  had  a  relatively  concise 
list  of  principles  to  guide  their  actions. 

Initial  searches  and  following  up  on  citations  led  to  the  following: 
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USAID,  the  U.S.  Government’s  lead  agency  for  international  development.  Their 
statement  was  called  “Nine  Principles  of  Development.” 

Paris  Declaration  on  Aid  Effectiveness,  a  meeting  of  senior  officials  from  donor  and 
developing-country  governments,  development  banks  and  international  agencies.  Their 
statement  was  entitled  the  “Paris  Declaration  on  Aid  Effectiveness.” 

International  Committee  of  the  Red  Cross  (ICRC),  a  humanitarian  organization  with 
special  status  among  governments;  it  is  recognized  in  the  Geneva  Conventions.  Its 
statement  is  known  as  the  “Code  of  Conduct.” 

Development  Assistance  Committee  (DAC)  of  the  Organization  for  Economic  Co¬ 
operation  and  Development  (OECD),  an  international  organization  of  30  governments, 
including  the  United  States,  whose  charter  is  “to  address  the  economic,  social  and 
environmental  challenges  of  globalization.”  Its  statement  was  the  “DAC  Principles  for 
Evaluation  of  Development  Assistance.” 

Sphere  Project,  “a  group  of  NGOs  and  the  Red  Cross  and  Red  Crescent  Movement 
[formed]  to  develop  a  set  of  universal  minimum  standards  in  core  areas  of  humanitarian 
response. . .”  This  group’s  findings  are  known  as  the  “Core  Standards.” 

Australian  Agency  for  International  Development  (AusAID),  Australia’s  lead  agency  for 
international  development.  They  have  published  a  statement  called  the  “Statement  of 
International  Development  Practice  Principles.” 

Interaction,  the  “largest  alliance  of  U.S. -based  international  non-governmental 
organizations  (NGOs),  with  more  than  190  members  working  in  every  developing 
country.”  They  published  the  “Principles  for  Effective  Assistance.” 
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3.6  Synthesis  Matrix 


"Strategic" 

"Operational"/"Tactical" 

USAID  "Nine  principles" 

USAID 

Paris  Declaration 

ICRC 

OECD-DAC 

Sphere  Project 

AusAID 

Interaction 

Ownership 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Capacity  building 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Sustainability 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Selectivity 

Yes 

Yes 

Yes 

Yes 

Assessment 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Results 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Partnership 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Flexibility 

Yes 

Yes 

Yes 

Yes 

Accountability 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Lack  of  Consensus 

Impartiality 

No 

No 

Yes 

Yes 

Yes 

No 

Yes 

Impartiality  defined:  Humanitarian  aid  is  not  a  partisan  or  political  act;  aid  is  given  based  on 
need  alone.  Why?  1.  Humanitarian  imperative;  2.  Maintains  access  to  those  in  need;  3.  Protects 
"neutrality"  of  aid  workers. 

Figure  5.  Synthesis  Matrix 

I  arranged  a  synthesis  matrix  that  displays  organizations  across  the  top  and  USAID’s  principles 
down  the  left-hand  side.  “Yes”  in  the  matrix  in  Figure  5  above  indicates  that  an  organization 
agrees  with  the  principle  on  that  line.  “No”  means  they  disagree,  and  a  blank  space  indicates  that 
an  organization’s  statement  did  not  mention  that  principle  explicitly. 

Additionally,  I  added  “impartiality”  to  the  left-hand  side  because  it  was  the  only  major  principled 
disagreement  between  organizations.  Organizations  associated  with  governments  felt  no 
obligation  to  be  impartial  in  conflicts  in  the  developing  world;  rather  they  could  and  would  use 
aid  to  affect  political  outcomes.  In  contrast,  private  organizations  are  motivated  by  a 
“humanitarian  imperative.”  They  believe  that  need  is  the  sole  criterion  for  aid,  and  they  attempt 
to  maintain  a  strict  neutrality  in  conflicts. 

The  credo  of  impartiality  can  limit  the  ability  of  private  organizations  to  work  with  the  DoD  in 
conflict  zones. 

The  first  three  items  on  the  list — ownership,  capacity  building,  and  sustainability — are  what  I 
will  call  the  “big  three.”  Development  organizations  universally  subscribe  to  these.  I  will  make 
the  case  that  these  are  essential  to  successful  development  outcomes,  and  that  a  development 
project  plan  should  explicitly  address  these  issues  to  maximize  the  likelihood  that  it  will  be 
effective. 
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The  remaining  principles  are  things  all  organizations  aspire  to  in  pursuit  of  organizational 
objectives. 

Andrew  Natsios,  a  former  Administrator  of  USAID,  agreed  with  my  analysis.  He  wrote,  in 
Parameters'. 

“ Ownership ,  capacity  building,  and  sustainability  form  an  iron  triad  of  principles 
underscoring  all  successful  and  enduring  development  and  reconstruction  projects.” 
(. Parameters ,  Autumn  2005,  p.  1 1) 

3.7  Effective  Projects  Worksheets 


A  theoretical  identification  of  the  “big  three”  is  not  sufficient  to  ensure  their  use  in  the  formation 
of  development  activity  plans.  DoD  practitioners  need  a  way  to  make  this  knowledge  useful.  I 
propose  the  mandatory  use  of  a  series  of  “worksheets”  containing  questions  that  should  answered 
or  issues  that  should  be  addressed  in  order  for  project  proposals  to  be  evaluated,  prioritized, 
selected,  funded,  and  executed. 

Collectively,  I  will  call  these  “Effective  Projects  Worksheets.”  Their  purpose  is  to  force 
practitioners  to  consider  the  “big  three,”  and  to  identify  and  mitigate  key  risks  to  the  “big  three.” 

Additionally,  military  organizations  have  long  made  explicit  the  purpose  of  any  activity  they 
undertake,  so  that  subordinates  can  clearly  understand  what  the  commander  wants.  Development 
activities  should  be  no  different.  The  explicit  statement  of  the  objective  makes  its  achievement 
more  likely. 

While  the  commander  is  likely  most  interested  in  influence  effects,  the  explicit  statement  of  the 
desired  development  effects  makes  their  achievement  more  likely  as  well.  Since  a  lasting 
development  effect  is  essential  for  a  lasting  influence  effect,  the  achievement  of  the  development 
effect  that  the  local  people  desire  is  essential  to  the  achievement  of  the  influence  effect  that  the 
commander  desires. 

Explicit  consideration  of  the  purposes  of  an  activity  and  the  “big  three”  via  an  “Effective 
Assistance  Worksheet”  would  ask  these  or  similar  questions: 

Purpose:  What  influence  effect  do  we  expect  (refer  to  campaign  objectives)?  What 
benefit  do  we  expect  to  accrue  to  local  beneficiaries  (development  effect)?  Do  we  want 
to  publicize  the  project?  Will  USG  and/or  host  nation  get  credit? 

Ownership:  Who  are  the  stakeholders?  How  have  they  been  engaged?  How  are  they 
active  in  planning  the  project?  How  will  beneficiaries  contribute  to  the  project  through 
their  own  resources? 

Capacity-building:  Are  local  capacities  improved  so  that  project  is  sustainable?  Can  local 
people  run  it  themselves?  Do  local  people  need  training?  How  will  this  happen? 

Sustainability:  Do  the  beneficiaries  have  the  will  and  capacity  to  maintain  the  project? 
How  will  they  do  this?  How  will  local  national  government  officials  support  the  project? 
Does  the  U.S.  need  to  fund  maintenance?  For  how  long?  How  does  the  project  interact 
with  other  USG  initiatives?  With  civil  society  initiatives? 
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The  next  several  paragraphs  detail  lists  of  questions  designed  to  ensure  project  design  identifies 
and  mitigates  risks  to  the  “big  three.”  These  questions  assist  in  assessing  and  addressing 
important  risks  to  achievement  of  either  development  or  influence  effects. 

Much  of  the  criticism  contained  in  the  literature  on  development  activities  in  conflict  zones 
reflects  the  ease  with  which  funds  or  materials  can  feed  corruption  or  be  diverted  due  to  criminal 
activity,  such  as  extortion  or  theft.  Additionally,  projects  can  have  unintended  economic 
consequences  by  impacting  markets,  businesses  or  consumers  in  unexpected  ways. 

Detennining  the  likelihood  and  severity  of  risk  inherent  in  these  types  of  unintended  effects  is 
critical  to  the  execution  of  a  project  that  will  have  a  net  positive  effect  among  the  target 
population.  Identification  of  risks  permits  mitigations  to  be  designed  into  the  project  plan  or  may 
require  cancellation  of  the  project. 

The  “Corruption,  Criminality,  and  Economic  Risk  Assessment”  questions  are  designed  to 
illuminate  such  risks: 

What  is  the  risk  that  project  will  feed  corruption  or  criminality? 

What  is  the  opportunity  of  local  officials  or  traditional  leaders  to  embezzle  from  the 
project? 

What  is  the  likelihood  insurgents  or  criminals  will  receive  protection  money? 

What  is  the  risk  of  materials  being  stolen  from  project  inventory? 

Will  local  people  be  subject  to  intimidation  or  violence? 

What  is  the  risk  the  project  will  be  converted  to  an  unintended  use? 

Will  the  project  damage  (“crowd-out”)  local  businesses? 

Will  local  procurement  of  materials  negatively  impact  local  markets  by  creating 
shortages  or  raising  prices? 

Will  the  project  disproportionately  benefit  a  private  individual  or  company  or  create 
unfair  competitive  advantage  for  an  individual  or  company? 

What  are  other  potential  unintended  effects  of  a  project?  Does  it  threaten  particular 
interest  groups?  Traditional  leaders?  Rival  groups?  Businesses?  Criminals? 

How  can  identified  risks  be  mitigated?  What  is  the  residual  risk? 

Additionally,  projects  may  have  unintended  effects  on  the  environment  that  may  lead  to  adverse 
effects  on  the  people  we  are  trying  to  benefit.  Some,  if  identified,  can  be  mitigated  easily. 

Others  may  require  fundamental  reconsideration  of  the  project. 

The  following  questions  represent  an  initial  attempt  at  development  of  a  simple  “Environmental 
Risk  Assessment”  that  may  be  sufficient  for  small-scale  projects.  The  list  could  possibly  be 
improved  by  someone  with  a  background  in  environmental  engineering. 

Will  the  project  enable  or  encourage  the  population  to  engage  in  environmentally 
unsustainable  practices,  e.g.,  exceed  carrying  capacities  for  rangeland;  change  hydrology 
or  water  use  patterns;  over-consume  energy  stocks  like  gas,  bio-mass,  or  electricity? 

Does  the  project  create  toxic  byproducts?  Can  this  be  mitigated? 
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Will  changes  in  people’s  behaviors  due  to  the  project  create  environmental  problems? 

Will  the  project  work  in  the  dry  and  rainy  seasons?  In  summer  and  winter? 

Does  project  design  avoid  creating  hazards  or  mitigate  likely  hazards,  e.g.,  does  the 
project  protect  a  newly  created  water  source  from  contamination? 

Planners  also  want  to  think  carefully  about  which  group  a  project  benefits,  directly  and 
indirectly,  and  which  groups  it  might  hurt,  and  also,  which  group  might  be  resentful  that  the 
neighbors  got  something  they  did  not. 

Planners  need  to  be  sure  that  the  beneficiaries  own  the  property  the  project  sits  on  so  that  they 
can  benefit  from  it  over  a  reasonable  project  lifespan. 

Planners  want  to  know  if  the  project  is  employing  local  firms  and  workers  or  importing  others 
from  outside  the  area,  and  confirm  that  local  firms,  if  contracted,  have  the  expertise  and 
equipment  needed  to  complete  the  project  satisfactorily. 

The  following  “Distributed  Benefits  Worksheet”  questions  address  such  issues: 

Who  owns  the  land  on  which  a  project  will  be  executed?  Does  land  title  need  to  be 
transferred?  To  whom? 

Are  there  other  property  rights  issues?  Water  rights?  Navigation?  Rights  of  way? 

Who  is  the  prime  contractor?  Sub-contractors?  Do  local  firms/groups/people  have  the 
capacity  (expertise  and  physical  ability)  to  do  the  work? 

How  much  local  labor  is  employed?  Is  there  opportunity  for  “sweat  equity”? 

What  materials,  equipment,  or  expertise  is  required  which  are  not  available  locally? 

Are  there  “losers”  if  the  project  is  executed?  Who  are  they?  Why?  Can  they  be  co-opted 
or  compensated? 

Does  the  project  compensate  beneficiaries  for  an  activity  they  have  done  themselves  in 
the  past?  That  is  generally  undesirable;  is  there  some  characteristic  of  that  activity  that 
would  make  it  desirable  to  compensate  it? 

Is  the  project  likely  to  create  or  encourage  dependency? 


These  questions  or  modifications  of  them  should  be  added  to  existing  project  nomination 
worksheets,  such  as  those  in  the  Overseas  Humanitarian  Assistance  Shared  Infonnation  System 
(OHASIS),  to  ensure  that  project  planners  consider  these  issues. 
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4  INTEGRATING  ASSESSMENT 

The  third  objective  sought  to  integrate  assessment  into  project  planning.  This  required  the 
development  of  tools  for  planning  and  implementing  assessment  as  an  integral  part  of  project 
planning,  execution,  and  follow-through. 

To  accomplish  this,  I  surveyed  literature  to  determine  the  key  elements  of  an  assessment  plan. 
These  key  elements  must  be  integrated  into  the  project  plan. 

4.1  Assessing  Progress 


CA  Methodology 


Assess 


Decide 


Develop  and 
Detect 


Deliver 


Evaluate 


Transition 


Proposed  Methodology 


Assessment  determines  needs. 


Civil  Affairs  Assessment 


Alternative  projects  and 
conflicting  needs  compete  for 
selection.  Most  effective  projects 
given  priority.  Identify  effects  on 
campaign. 


Effective  Project 
Worksheets 


Selected  project  plan(s)  fully 
developed. 


Project  executed. 


Assessment  Plan 
Statement  of  Work 


Project  Execution 


Assessed  during/after  execution. 


Assessment 


Got  Ownership?  [Hand  off  to  HNG*  or  beneficiaries 

"Host  nation  government 


Figure  6.  Assessing  Progress 

Integration  of  the  assessment  plan  into  the  project  plan  not  only  enhances  the  formal  assessment 
of  the  project,  but  also  improves  the  project  plan  by  improving  the  understanding  of  the 
environment  and  the  likely  second-order  effects  that  stem  from  project  execution.  This  section 
of  the  report  attempts  to  refine  the  portion  of  the  CA  methodology  that  is  contained  in  the  red 
box  in  Figure  6  above. 

One  key  part  of  the  project  plan,  particularly  if  the  plan  requires  contracted  labor  or  expertise,  is 
the  statement  of  work  (SOW).  It  is  the  core  of  the  contract  because  it  sets  out  the  standards  for 
the  work  being  done,  and  therefore,  constitutes  the  contractor’s  obligations  to  the  funder. 

A  poorly-specified  SOW  leaves  out  critical  quality  standards,  and  leaves  the  quality  of  work 
completely  up  to  the  contractor.  It  is  impossible  for  the  funder  to  say  later  that  the  work  is  not  to 
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standard  if  the  standards  are  not  spelled  out.  Poor  specifications  are  why  the  United  States  has 
paid  for  a  number  of  shoddy  and  unsustainable  projects  in  the  past  decade. 

The  International  Building  Code,  published  by  the  International  Code  Council,  can  serve  as  the 
basis  of  standards  that  are  written  into  statements  of  work.  The  key  is  to  ensure  safe  and  well- 
built  projects  in  accordance  with  standards,  but  not  to  build  things  that  cannot  be  maintained 
locally. 

A  recent  Naval  Postgraduate  School  thesis,  entitled  Sustainable  Construction  in  Afghanistan 
(2010)  by  LCDR  Legena  Malan,  argues  that  construction  in  Afghanistan  is  better  if  it  uses  local 
materials  and  techniques  and  conforms  to  local  customs  so  that  it  is  both  acceptable  to  the  people 
and  maintainable  by  the  people.  Project  planners  must  blend  international  standards  with  local 
techniques  for  optimal  results. 

It  is  useful  at  this  point  to  make  several  additional  observations  about  assessment: 

“Assessment”  in  this  context  refers  to  campaign  assessment-  determining  the 
effectiveness  of  activities  in  achieving  goals.  Development  agencies  refer  to  this  as 
“evaluation”  or  “monitoring  and  evaluation.” 

The  purpose  of  assessment  (evaluation)  is  to  detennine  if  objectives  are  being  met, 
internalize  lessons  learned,  and  adjust  project  activities  to  be  more  effective  or  more 
appropriate  to  a  changing  situation.  Assessment  also  provides  accountability  to  donors, 
local  national  authorities,  and  beneficiaries. 

Assessment  is  best  when  planned  in  parallel  with  project  planning;  the  assessment  plan  is 
an  integral  part  of  the  project  plan.  Assessment  is  an  ongoing  process  to  ensure  a  project 
delivers  the  intended  benefits  over  a  reasonable  lifespan. 

Quantitative  measures  are  desirable,  but  activities  that  are  most  “transformational”  are 
also  the  hardest  to  measure  quantitatively  (Natsios,  2010). 

-  Assessment  depends  upon  a  manageable  set  of  measures  of  perfonnance  (MoPs)  and 
measures  of  effectiveness  (MoEs),  and  will  likely  include  relevant  local  or  international 
standards  of  quality  (e.g.,  water  quality  for  a  drinking  water  project). 
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4.2  A  Project  Assessment  Model 


Project  Assessment 


“ Measuring  Performance  Measuring  Effectiveness 

Planning  Start-up  (MOPs)  (MOEs) 


•  Resource: 

—  Haims  et  al.  (RAND*)  (2011).  Developing  a  Prototype  Handbook  for 
Monitoring  and  Evaluating  DoD  Humanitarian  Assistance  Projects. 

•  A  model  of  project  assessment  developed  for  DoD  HA**  projects. 

•  Report  is  relatively  short  and  contains  a  lot  of  information  on  how 
to  nest  project  objectives  under  campaign  objectives. 


*The  RAND  Corporation 
“Humanitarian  Assistance 


Figure  7.  A  Project  Assessment  Model 

A  mental  model  serves  to  organize  our  thinking  about  a  particular  task.  Such  a  model  is 
beneficial  for  designing  and  assessing  development  projects.  The  RAND  Corporation  has 
designed  a  model  to  organize  project  assessment. 
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4.3  Assessment  Planning  in  Parallel 


•  Assessment  planning  is  integral  to  project  planning. 

•  RAND’s  model  is  designed  to  be  parallel  to  a  “Project  Cycle 
Model”  contained  in  the  same  reference. 


Figure  8.  Assessment  Planning  in  Parallel 

Since  the  assessment  and  the  project  are  planned  together,  the  assessment  model  should  be  used 
in  parallel  with  a  project  cycle  model  that  organizes  our  thinking  about  project  planning  and 
design.  RAND  has  also  developed  a  project  cycle  model  that  works  in  parallel  with  their 
assessment  model. 

4.4  Assessment  Template  -  Before/During  Execution 


In  addition  to  RAND’s  work,  I  have  identified  some  questions  that  planners  should  answer 
before  beginning  the  project  and  during  project  execution.  They  are: 

Is  there  a  relevant  baseline  of  conditions  existing  prior  to  project  execution? 

How  have  beneficiaries  and  stakeholders  been  included  in  project  design  and  planning? 

Are  they  willing  and  able  to  contribute  meaningfully  to  the  project? 

What  local  codes  or  international  standards  are  appropriate  and  should  be  part  of  the 
statement  of  work? 

Do  we  have  the  expertise  and  access  to  ensure  the  contractor  meets  the  standards 
contained  in  the  statement  of  work?  Where  can  we  obtain  this  expertise? 

Are  project  management  indicators  being  met  (see  “core  indicators”  below)? 
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The  second  and  third  questions  ensure  beneficiary  ownership  of  the  project.  Planning  it  makes  it 
theirs  and  makes  it  meet  their  needs.  Contribution  to  it  means  they  really  want  it.  It  is  the 
crucial  test  of  ownership. 

4.5  Assessment  Template  -  After  Execution 


Other  questions  are  appropriate  after  the  project  is  complete.  These  questions  also  touch  on 
ownership,  but  they  are  designed  to  confirm  the  sustainability  of  the  project  and  to  ensure  local 
national  capacities  were  built.  They  also  capture  lessons  learned  and  directly  test  whether  the 
desired  effects  were  achieved. 

Does  the  project  deliver  intended  benefits  30  days  after  completion?  90  days?  1  year? 

Is  the  project  being  used  as  intended? 

Are  local  people  managing,  maintaining,  and  sustaining  the  project  themselves?  How? 

(This  might  be  a  lesson  learned  or  a  best  practice. . .) 

How  have  local  capacities  improved? 

How  do  actual  benefits  compare  to  anticipated  benefits?  Are  the  beneficiaries  satisfied? 

How  do  we  know  they  are? 

Did  the  project  achieve  its  purposes  (development  and  influence  effects)?  How  do  we 

know? 

What  additional  MoPs  and  MoEs  provide  evidence  that  we  were  successful?  How  will 

these  be  collected? 
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5  METRICS  MENUS 

One  of  the  study  goals  was  to  develop  a  series  of  “metrics  menus”  from  which  a  project  planner 
could  select  several  metrics  to  assess  the  success  of  the  project. 

I  found  that  the  issues  surrounding  the  most  common  DoD  project  types  are  not  quite  that 
simple.  Selecting,  collecting,  and  evaluating  indicators  and  metrics  is  a  complicated  process  that 
is  best  done  by  people  with  expertise  in  a  specific  project  type. 

A  “generalist”  CA  soldier  cannot  do  the  most  effective  work  without  some  self-study  on  the 
project  types  that  are  under  consideration.  Therefore,  additional  information  resources  are 
required  for  most  common  project  types.  These  contain  metrics  that  may  be  selected  as  MoPs  or 
MoEs,  and  some  are  identified  below. 

Participatory  monitoring  and  evaluation  (UN  Population  Fund,  2004)  may  serve  as  a  useful 
model  for  improving  assessment,  developing  ownership,  building  capacity,  and  ensuring 
sustainability.  In  this  model,  beneficiaries  assume  responsibility  for  various  assessment  tasks  in 
lieu  of  CA  soldiers. 

Despite  the  great  variety  of  problems  encountered  by  practitioners  in  the  field,  my  literature 
review  identified  some  “core  indicators”  (Haims  et  al. ,  2011)  that  apply  to  most  projects, 
developed  metrics  menus  for  different  project  types,  and  identified  additional  resources  for 
project  planners  to  consult. 

5.1  Core  Indicators 


RAND  (Haims  et  al.,  2011)  developed  a  list  of  process  metrics  (“core  indicators”)  that  are 
common  to  all  project  types.  These  “core  indicators”  are  used  to  monitor  the  work  as  it  moves 
forward  to  ensure  that  is  proceeding  efficiently. 

Some  examples  include: 

Assessment/sustainment  plans  complete  before  execution? 

Project  completed  on  time? 

Project  completed  under  budget? 

Project  coordinated  with  relevant  USG  agencies? 

Baseline  assessment  completed? 

Project  implemented  in  collaboration  with  local  civilians? 

Number  of  host  nation  civilians  trained? 

Host  nation  civilians  operate/maintain  project  after  execution? 

Project  transfers  knowledge  to  host  nation  citizens? 

Project  information  recognizing  DoD  and  host  nation  government  disseminated  in  local 
community? 

The  responses  are  often  “yes/no,”  a  Likert  scale,  or  a  count. 
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5.2  Most  Common  DoD  Projects 

RAND  also  identified  the  most  common  humanitarian  assistance  projects  funded  by  the  DoD. 
These  include: 

Water  and  sanitation  projects. 

Health  infrastructure  projects. 

Non-health  infrastructure  projects. 

Health  services  projects. 

Disaster-related  projects. 

RAND  also  developed  lists  of  metrics  specifically  related  to  these  project  types,  but  these 
metrics  do  NOT  capture  the  complexity  of  many  of  the  issues;  they  are  only  a  good  start.  So  I 
have  tried  to  expand  upon  these  lists  below. 

5.3  Water  and  Sanitation  Overview 

Water  and  sanitation  are  critically  linked  because  poor  hygiene  is  the  most  common  cause  of 
water  contamination. 

Most  DoD  water  projects  involve  building  wells  or  small-scale  water  treatment.  Water  projects 
also  may  consider  building  latrines  and  solid  waste  disposal  or  drainage  improvement,  both  of 
which  impact  water  quality. 

Delivery  of  good  quality  drinking  water  is  a  complex  problem. 

Baseline  assessment  seeks  to  understand: 
o  Current  practices  for  obtaining  water, 
o  Current  water  quality, 
o  Water  use  patterns. 

o  Duration  of  need — emergency  only  or  sustained  use? 

Water  sourcing — surface  or  ground  water? 

Will  disinfection  be  required?  How? 

Capacity  building — sanitation  practices,  local  maintenance,  preventing  contamination  of 
drinking  water  at  the  source  and  at  the  point  of  use. 

5.4  Water  and  Sanitation  Metrics 

Some  sample  metrics  for  water  projects  include: 

-  Number  of  potable  sources  created. 

Source  improved  in  accordance  with  relevant  international  or  local  standards. 

Source  improved  to  allow  local  maintenance. 

Source  improvement  accounts  for  local  practices,  e.g.,  bathing,  watering  livestock. 
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Run-off  does  not  cause  additional  problems,  e.g.,  mosquito  breeding  area. 

Number  of  people  obtaining  water  daily/number  of  people  within  500  meters. 

Queuing  time  at  source. 

Volume  (recommended  amounts  for  different  purposes;  see  Sphere  Handbook.) 

Flow  rate. 

-  Turbidity. 

Fecal  contamination  (source  and  point  of  use). 

Chlorine  residue  after  disinfection. 

Chemical  contamination. 

Locals  maintain  facilities? 

Locals  monitor  water  quality? 

An  additional  resource  for  assessment  of  the  effectiveness  of  water  projects  is  the  World  Health 
Organization’s  “Core  Questions  on  Drinking-water  and  Sanitation  of  Households”  survey.  It  is 
designed  to  assess  improvements  in  families’  water  and  sanitation  access  and  practices.  It  is  a 
fully  developed  survey  (in  English  at  least). 

Some  sample  questions  include: 

What  is  the  main  source  of  drinking  water  for  your  household? 

How  long  does  it  take  to  go  to  the  water  source,  get  water,  and  come  back? 

Who  usually  goes  to  the  water  source  to  fetch  water? 

It  is  available  at: 

http://www.who.int/water  sanitation  health/monitoring/oms  brochure  core  questionsfinal2460 

8.pdf 

5.5  Health  and  Non-Health  Infrastructure  Overview 

Most  DoD  health  projects  involve  construction  or  refurbishment  of  clinics  or  hospitals. 

There  are  many  possible  project  objectives  due  to  the  large  number  of  different  health  services 
that  could  be  delivered  at  a  given  facility,  so  there  are  many  possible  metrics. 

Examples  of  non-health  infrastructure  projects  include  local  government  facilities  (e.g.,  police 
stations,  council  buildings),  power  plants,  schools,  orphanages,  women’s  centers,  or  roads. 

Again,  the  diversity  of  project  objectives  yields  a  large  number  of  potential  metrics. 

5.6  Infrastructure  Metrics 

Health  and  non-health  infrastructure  metrics  are  very  similar.  Some  sample  metrics  include: 
Project  is  complete  on  time  and  under  budget. 

Facility  is  open. 

Facility  is  used  as  intended. 
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Facility  is  built  to  allow  local  maintenance. 

Facility  has  trained  staff. 

Number  of  people  using  facility/number  living  within  5  kilometers  or  1  hour. 

Number  of  people  trained  in  construction. 

Reduced  time  to  get  to  markets  (road  project). 

Additional  metrics  may  track  the  effectiveness  of  services  provided  at  the  project  building. 

5.7  Health  Services  Overview 

The  development  of  sustainable  health  services  is  fundamentally  an  exercise  in  capacity¬ 
building. 

Often  DoD  health  services  projects  are  related  to  emergencies  or  disasters,  to  the  direct  provision 
of  services  in  high  need  areas,  or  to  the  training  of  local  medical  personnel. 

RAND  expects  the  number  of  health  services  projects  to  increase. 

5.8  Health  Services  Metrics 

The  large  variety  of  possible  health  services  objectives  makes  identification  of  metrics  difficult. 
Nevertheless  some  sample  metrics  are: 

Children’s  weights  related  to  their  heights. 

Infants’  birth  weights. 

Rate  of  immunization. 

Age-specific  mortality  rates. 

Disease-specific  mortality  rates. 

Infant  and  maternal  mortality  rates. 

Health  services  are  accessible  to  vulnerable  members  of  the  population. 

Birth  attendance  by  trained  personnel. 

Availability  of  key  drugs. 

Ratio  of  population  to  different  kinds  of  health  workers. 

Additional  metrics  can  be  found  in  the  Sphere  Project  Handbook’s  “health  assessment  checklist” 
and  tools  are  at  Sphere’s  “sample  weekly  surveillance  reporting.” 

5.9  Disaster-related  Projects  Overview 

The  large  number  of  possible  disasters  makes  disaster  planning  and  disaster-related  project 
assessment  difficult. 

Disaster  aid  often  overlaps  with  one  or  more  of  the  previously  discussed  categories  of 
assistance — water  and  sanitation,  infrastructure,  and  health  services — or  it  is  related  to  food  and 
nutrition  or  shelter. 
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5.10  Disaster-related  Metrics 

Despite  the  wide  variety  of  disaster  scenarios,  some  sample  metrics  include: 
-  Number  of  structures  rebuilt,  relocated,  or  retrofitted. 

Key  structures  (hospitals,  schools)  have  power  and  water. 

Displaced  persons  are  informed. 

Number  of  volunteers  trained  to  implement  disaster  plan. 

Number  of  personnel  trained  to  maintain  disaster-resistant  structures. 
Crude  mortality  rate. 

Water  available  per  person. 

Food  available  >2100  calories  per  day. 

Shelter  space  >3.5  square  meters  per  person. 

Number  of  toilets  available. 


DALAM 


METRICS  MENUS  •  27 


CAA-20 12049 


(THIS  PAGE  INTENTIONALLY  LEFT  BLANK) 


28  •  METRICS  MENUS 


DALAM 


CAA-20 12049 


6  STUDY  QUALITY  AND  INSIGHTS 

6.1  Study  Quality 


According  to  Lincoln  and  Guba  (1985),  qualitative  research  should  have  the  following  attributes: 

Credibility:  Are  the  results  right? 

o  The  synthetic  nature  of  the  inquiry  and  the  strength  of  the  consensus  indicate 
credibility. 

Transferability:  Can  the  results  be  used  in  another  context? 

o  The  wide  dispersion  of  development  organizations’  operations  suggests 
transferability. 

Dependability:  Are  results  reliable  over  time? 

o  Common  assertions  that  lesser-developed  people  live  in  a  manner  similar  to 
generations  of  ancestors  suggest  dependability. 

Confirmability  or  auditability:  Is  the  method  transparent? 

o  The  wide  citation  of  disparate  sources,  most  of  which  are  freely  available  online, 
indicate  auditability. 

6.2  Insights 


The  study  identified  several  insights: 

Considerable  consensus  exists  in  the  development  community  on  principles  that  make 
assistance  more  effective. 

Military  organizations  doing  similar  work  can  improve  effectiveness  by  applying  these 
principles. 

Assessment  is  more  easily  and  better  accomplished  when  planned  concurrently  with 
project  planning. 

The  inclusion  of  these  findings  in  the  next  revision  of  project  nomination  forms  would 
improve  DoD  project  effectiveness. 
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APPENDIX  A  PROJECT  CONTRIBUTORS 

A-l  PROJECT  TEAM 

Project  Director:  Adam  Shilling,  PhD. 

Other  Contributors:  Center  for  Anny  Analysis  (CAA)  reviewers. 

A-l  PRODUCT  REVIEWERS 

Mr.  Russell  Pritchard,  Quality  Assurance 
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APPENDIX  B  REQUEST  FOR  ANALYTICAL  SUPPORT 


REQUEST  FOR  ANALYTICAL  SUPPORT 


Performing  Division:  OA 


Acronym:  DALAM 


Account  Number:  2012049 


FY:  2012 


Start  Date:  10-Nov-ll  Est  Compl  Date:  15-Mar-12 


Title:  Development  Activities  Locator  and  Assessment  Method  (DALAM) 
Category:  Current  Operations  Analysis 
Sponsor  (Le.,  DCS-G3)  Name:  CJTF-HOA 


Method:  In-house 


Office  Symbol:  CJTF-HOA  CJ5 


Phone:  (311)  824-4091 


E-Mail: 


POC:  CAPT  (USN)  Peter  Haynes 


Resource  Estimates: 


a.  Estimated  PSM: 


b.  Estimated  Funds: 


$0.00 


Models  to  be  Used:  None 


Product:  Briefing  and  Report 


Description/A  bstract: 

Combined  Joint  Task  Force  -  Horn  of  Africa  (CfIT-HOA)  seeks  to  improve  the  methods  by  which  they  emptoy  Civil  Affairs  (CA) 
teams  assigned  to  them  and  assess  the  effects  of  these  teams'  activities. 


Study  Director/POC  Signature: 


Phone:  703-806-6632 


Study  Director/POC:  Dr.  Adam  P  Shilling 


PART  2 


ISackground/Statement  of  Problem: 

CJTF-HOA  is  not  satisfied  with  the  methods  they  currently  use  to  plan  for  and  assess  development  and  relief  activities.  The 
command  is  concerned  that  these  activities,  typically  performed  by  attached  CA  teams,  are  not  as  effective  as  they  could  be  and  that 
assessment  of  these  activities  can  be  improved. 


Scope: 

The  sponsor  envisions  the  study  having  three  core  objectives: 

1.  A  method  to  determine  where  CA  teams  should  be  placed. 

2.  A  method  to  determine  which  specific  CA  activities  for  a  given  area  are  likely  to  be  most  effective. 

3.  A  method  to  assess  the  effectiveness  of  CA  teams’  activities. 


Issues: 


Milestones: 


CAA  Division  Chief  Signature: 


Date 


CAA  Division  Chief  Name:  COL  Steven  A  Stoddard 


Signature 


Sponsor  Concurrence  Signature : 


Date 


Sponsor  Name  (COL/DA  Div  Chief/GO/SES): 


Print  Date:  26-Od-U 
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APPENDIX  C  RESOURCES 

C-l  Selected  References 


The  list  below  contains  a  sample  of  the  literature  consulted  in  pursuit  of  study  objectives.  The 
reader  will  find  additional  resources  for  practitioners  in  the  sections  below. 

Australian  Agency  for  International  Development  (2009).  Engaging  Not-for-Profit 
Organisations :  Statement  of  International  Development  Practice. 

Development  Assistance  Committee  (1991).  Principles  for  Evaluation  of  Development 
Assistance.  Organisation  for  Economic  Co-operation  and  Development. 

High  Level  Forum  (2005).  Paris  Declaration  on  Aid  Effectiveness:  Ownership, 
Harmonisation,  Alignment,  Results  and  Mutual  Accountability. 

Interaction  (no  date).  Principles  for  Effective  Assistance. 

International  Committee  of  the  Red  Cross  (1996).  The  Code  of  Conduct  for  the 
International  Red  Cross  and  Red  Crescent  Movement  and  NGOs  in  Disaster  Relief. 

Lincoln,  Y.  S.,  &  Guba,  E.  A.  (1985).  Naturalistic  Inquiry.  Beverly  Hills:  Sage. 

Natsios,  A.  (2005).  The  Nine  Principles  of  Reconstruction  and  Development. 
Parameters,  Autumn  2005.  pp.  4-20. 

Natsios,  A.  (2010).  The  Clash  of  the  Counter-Bureaucracy  and  Development.  Center  for 
Global  Development  Essay. 

Sphere  Project  (2011).  Humanitarian  Charter  and  Minimum  Standards  in  Humanitarian 
Response  (“ Sphere  Handbook"). 

U.S.  Agency  for  International  Development  (2011).  The  Nine  Principles  of 
Development. 

C-2  Comprehensive  Practitioners’  Resources 


There  are  several  resources  that  are  comprehensive  in  nature.  They  address  a  number  of 
development  areas,  and  provide  a  basic  understanding  of  the  issues  in  those  areas,  or  contain 
information  appropriate  to  all  project  types. 

I  have  identified  five  such  resources.  Four  of  them  are  available  in  electronic  format  at  no  cost 
online.  Only  the  Oxfam  Handbook  requires  purchase,  but  used  copies  are  available  for  little 
money. 

Haims  et  al.  (20 11).  Developing  a  Prototype  Handbook  for  Monitoring  and  Evaluating 
DoD  Humanitarian  Assistance  Projects.  RAND. 

o  Relatively  brief  discussion  of  the  importance  of  assessment. 

o  Develops  a  conceptual  model  to  drive  assessment,  and  identifies  “core  indicators” 
that  gauge  a  project’s  progress  during  execution. 

o  A  short  list  of  metrics  for  each  of  the  five  common  project  types. 
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Oxfam  (1995).  The  Oxfam  Handbook  of  Development  and  Relief  (in  three  volumes). 

o  A  bewilderingly  comprehensive  introduction  to  the  issues  of  many,  many  facets  of 
relief  and  development. 

o  Too  many  subjects  to  list. 

o  Chapters  end  with  “Key  Questions”  that  illuminate  important  issues  and  provide 
seed  for  metrics. 

The  Sphere  Project  (2011).  Humanitarian  Charter  and  Minimum  Standards  in 

Humanitarian  Response  (commonly  called  the  “Sphere  Handbook”). 

o  Sphere  Project,  a  consortium  of  NGOs,  “identified  a  set  of  minimum  standards  in 
key  life-saving  sectors,”  including  water  supply,  sanitation  and  hygiene 
promotion;  food  security  and  nutrition;  shelter,  settlement  and  non-food  items; 
and  health  action. 

o  Chapters  are  organized  into  Minimum  Standards,  Key  Actions,  Key  Indicators, 
and  Guidance  Notes  that  contain  a  wealth  of  possible  metrics  and  notes  on  how  to 
use  them  to  improve  service  delivery. 

o  Each  chapter  offers  a  “Further  Reading”  section  with  many  more  resources. 

United  Nations  Population  Fund  (2004).  Stakeholder  Participation  and  Monitoring. 

Programme  Manager’s  Monitoring  and  Evaluation  Toolkit. 

o  Explains  the  benefits  of  participatory  assessment. 

o  Available:  http://www.unfpa.org/monitoring/toolkit/stakeholder. pdf 

USAID  (2005).  Field  Operating  Guide  v.  4.0. 

o  Contains  detailed  information  on  conducting  baseline  assessments,  and  a  series  of 
“Assessment  Checklists”  that  ask  relevant  questions  that  might  serve  as  bases  for 
metrics. 

o  The  chapter  titled  “Information  on  Populations  at  Risk”  contains  a  variety  of 
possible  metrics  expressed  in  paragraphs. 

o  Contains  infonnation  on  water,  food  and  nutrition,  health,  shelter  and  settlements, 
sanitation  and  environmental  health,  and  agriculture  and  livestock. 

C-3  Water  and  Sanitation  Resources 


WaterAid  UK  (no  date).  Technology  Notes. 

o  A  brief  introduction  to  water  provision  issues. 

o  http://www.wateraid.org/uk/what  we  do/sustainable  technologies/technology  no 

tes/default.asp 

World  Health  Organization  (2011).  Guidelines  for  Drinking-water  Quality,  4th  Ed. 

o  A  comprehensive  reference  (>500  pages)  on  the  issues  surrounding  the  provision 
of  quality  drinking  water. 
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World  Health  Organization  (2006).  Core  Questions  on  Drinking-water  and  Sanitation  for 
Household  Surveys. 

o  A  fully-developed  survey  for  assessing  the  effectiveness  of  drinking 
water/sanitation  projects. 

The  World  Health  Organization  (WHO)  provides  many  references  on  water  and 
sanitation: 

o  hhp://www.who.int/water  sanitation  health/en/ 

The  U.S.  Environmental  Protection  Agency  also  provides  useful  info: 
o  http://water.epa.gov/ 

C-4  Infrastructure  Resources 

USAID  (2005).  Field  Operating  Guide  v.  4.0. 

Local  national  or  U.S.  building  codes. 

The  “International  Building  Code.” 

Appropriate  technical  expertise  from  engineers,  architects,  contractors,  or  skilled 
tradesmen. 

C-5  Health  Services  Resources 

Appropriate  trained  health  care  practitioners — medics,  nurses,  doctors,  therapists,  etc. 

U.S.  Department  of  Health  and  Human  Services: 

o  http://heahhypeople.gov/2020/default.aspx  and  click  tab  for  “leading  health 
indicators.” 

Center  for  Disease  Control: 

o  http://www.cdc.gov  and  choose  from  “health  and  safety  topics”  on  the  home  page. 
Werner  et  al.  (2010).  Where  There  Is  No  Doctor:  A  Village  Health  Care  Handbook. 

o  hhp://weblife.org/pdf/where  there  is  no  doctor.pdf 
Dickson  (2010).  Where  There  Is  No  Dentist: 

o  hhp://weblife.org/pdf/where  there  is  no  dentist.pdf 

C-6  Disaster-related  Resources 

Other  references  previously  mentioned  regarding  water,  sanitation,  health  care,  or  infrastructure. 
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